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Over the past 15 years, innovative and rigorous research has identified and evaluated effec- 
tive practices to support students’ success in college. This research has produced a growing 
body of evidence for strategies proven to address the critical barriers to attaining a post- 
secondary credential or degree. These strategies include reforms to instruction, financial 
aid, advising and nonacademic support systems, and career and technical education.* 


Additionally, states’ and institutions’ increasing adoption of evidence-based policy- 

making represents a major shift toward prioritizing and integrating research findings and 

data in higher education policy decisions. Further, the federal government has expressed 

a renewed and increased interest in investing in higher education, as demonstrated, for 

example, by the funding allocated for colleges in the American Rescue Plan to retain and 

\ support students affected by disruptions associated with the COVID-19 pandemic.* Simi- 

iS SHEEO larly, momentum has gathered behind the proposed investments in evidence-based student 
rg arin each success strategies (known as the College Completion Fund) in the American Families 
Plan.* However, fundamental misalignments remain between the domains of research and 
policy despite this increased attention, limiting the influence that data and research have 
on policy and practice. 


I ] laine The Evidence to Action project (2019-2021), led by MDRC and the State Higher Educa- 
Bry statis esa ae ere tion Executive Officers Association (SHEEO) and supported by Arnold Ventures, initi- 
. ated a body of work designed to disentangle the barriers that exist between research and 


state-level higher education policy and partnered with state higher education agencies 


to develop solutions. As the final step in the Evidence to Action project, this brief proposes prom- 
ising strategies found to address the factors that state-level postsecondary leaders have observed to 
have the most impact on evidence-based policymaking in higher education. This brief also provides 
recommendations and resources for state- and system-level decision-makers advocating for greater 
investment in evidence-based policymaking as well as for researchers and intermediary organiza- 
tions hoping to produce relevant research that they can translate into policy. 


EVIDENCE TO ACTION PROJECT OVERVIEW 


For the Evidence to Action project, MDRC and SHEEO partnered with four agencies across three 
states: the Indiana Commission for Higher Education (CHE), the Montana Office of the Commis- 
sioner of Higher Education (OCHE), the State Council of Higher Education for Virginia (SCHEV), 
and the Virginia Community College System (VCCS). The research team interviewed staff within 
these agencies and other key decision-makers in higher education across Indiana, Montana, and 
Virginia to understand how research and data were used—or not used—to identify policy priori- 

ties, inform policy design and adoption, and implement and later refine policy within the domains 
relevant to those states (for example, financial aid, developmental education, and career and technical 
education). Afterward, the research team summarized the findings in state-specific memos and con- 
vened the four state agencies to discuss common themes and emerging opportunities. 


The states of Indiana, Montana, and Virginia were selected to take part in the project because of 
their broad relevance for other states: They represent diverse policymaking contexts and experiences, 
stemming from the size and demographics of their populations, their cultures, their dominant polit- 
ical ideologies, and their higher education governance structures. Each agency’s approach to using 
research and data to inform policymaking is driven by these contexts: For instance, postsecondary 
institutions within the Montana University System and VCCS receive state funding through their 
respective system offices, which serves as an additional avenue for OCHE and VCCS to influence how 
those colleges operate. Additionally, each state agency has invested funding, resources, and support 
systems for institutions to improve the integration of data and research with policymaking—for 
example, each agency has developed robust systems and policies to collect and analyze data from 
across the state. Findings from experiences and perspectives shared by the state postsecondary lead- 
ers who participated in Evidence to Action inform recommendations in this brief. 


WHY IS EVIDENCE-BASED POLICYMAKING IMPORTANT? 


A state’s values and priorities for higher education may be defined by its culture, economic devel- 
opment goals, available resources, and other factors. Well-designed research plays an important 
role in supporting its priorities as well: It provides insight into the systemic and structural barriers 
that impede students’ progress through postsecondary education, highlights possible solutions to 
overcome those barriers, and minimizes the biases that may be introduced by relying on personal 


Evidence to Action 2 


experience alone. Rigorous research, particularly experimental studies like randomized controlled 
trials, builds confidence in the effectiveness of promising policies and practices and also contributes 
to an overall evidence base built on the strategies proven to support college students’ success. Addi- 
tionally, descriptive data collected and analyzed locally, qualitative research, and cost studies are 
other resources decision-makers can use to determine whether a strategy meets the needs of their 
communities and to secure the appropriate funding and resource investments required for imple- 
mentation. Assessing whether and how to use research to inform policy can be difficult: For a tool kit 
on assessing the quality of research and leveraging the different types of research that inform policy- 


making, see the first of the two supplements to this brief, “How Do I Assess the Value and Quality of 
Research?” 


Data and research are instrumental in addressing the unique needs of different communities and 
increasing equity in postsecondary education. Disaggregating and analyzing outcomes data by rele- 
vant populations and characteristics (such as socioeconomic status, race and ethnicity, and gender) 
can identify disparities in access or achievement. Qualitative research—in the form of interviews, 
focus groups, and observations—can enable students, colleges, community members, and other 
stakeholders to share their experiences and perceptions of the systemic or structural barriers hin- 
dering students’ progress in education. And finally, including subgroups within the analysis of more 
rigorous studies can help answer questions about which strategies do and do not work for various 
populations of interest. 


Many organizations have proposed frameworks for evidence-based policymaking to support poli- 
cymakers’ increased interest in the use of data and research.® Using these frameworks and interview 
findings, the Evidence to Action research team synthesized five foundational activities that are inte- 
gral to evidence-based policymaking in postsecondary education: 


1 Bring together a diverse and inclusive group of stakeholders to develop a shared vision and goals.’ 


2 Build or expand the policies, systems, and practices needed to collect, analyze, and communi- 
cate institutional or state data. 


3 Increase the capacity for stakeholders within the organization or across the state to use research 
and data. 


4 Create processes that center continuous improvement, assessment, and data-informed revision 
as policies are implemented. 


5 Incentivize further research by aligning it with the relevant policy levers in postsecondary edu- 
cation, such as budgeting or legislation. 


For more information about the critical actions required to support evidence-based policymaking, 


« 


see the second supplement to this brief, “Foundational Activities for Evidence-Based Policymaking in 
Postsecondary Education.” 
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FACTORS AFFECTING THE USE OF RESEARCH IN POLICYMAKING 


Despite the increased interest in evidence-based policymaking, fundamental challenges remain 

in using data and research to inform policy. Based on analyses of these barriers and discussions 
with the state agency partners in Indiana, Montana, and Virginia, the Evidence to Action research 
team identified the four most significant yet actionable barriers and promising strategies to address 
these barriers.® 


Barrier 1: Key decision-makers tend to have limited time and energy for engaging with research. 
Policymaking often takes place in a fast-paced environment, limiting stakeholders’ time to engage 
with it or to digest research related to their policy goals. The volume and complexity of the available 
research literature also creates difficulties. Additionally, many interview participants voiced that they 
or their colleagues have limited expertise or comfort with interpreting data or research studies and 
with assessing research studies’ quality or usefulness for the state. 


Promising Solutions: Box 1 summarizes promising solutions to address Barrier 1. Representatives 
from SCHEV and OCHE recommended planning ahead and setting clear priorities. When stake- 
holders agree to shared goals before policy opportunities emerge, they have more time to find and use 
relevant research to inform policy ideas. 


BOX 1 


PROMISING SOLUTIONS TO FACILITATE POLICYMAKERS’ 
ENGAGEMENT WITH RESEARCH 


[_] Develop a learning agenda that includes priorities and key research questions, and part- 
ner with researchers as needed to answer those questions. 


[_] Hire and designate staff with research expertise to prioritize using research and data to 
inform policymaking. 


[_] Provide training and professional development in research literacy for key stakeholders. 


[_] Use interns, graduate fellows, and local researchers to help assess the applicability and 
relevance of research and conduct analyses with internal data. 


[_] Formalize the use of research and data in policymaking. 


[_] Researchers: Make research findings more accessible. 


As one interview participant suggested: 
[Other policymakers] will have one really good idea in one meeting, and then they get 


another really good idea and try to put that into place. Maybe because we are so small, we've 
been really disciplined in moving forward policies that really matter, and committing to 
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them. ... We hit pause [each] summer and open up the dialogue about the most important 
areas. And then [we determine] the most important elements that we need to focus on in 
each and every one of our conversations throughout the year. 


As noted in Box 1, hiring or training staff members can increase your organization’s collective 
capacity to engage with research. However, when resources are limited, graduate fellows or academic 
researchers can, for example, scan and summarize research that aligns with the state’s priorities. 


Barrier 2: The research is not necessarily credible or persuasive to some audiences. Many inter- 
view participants emphasized that they found anecdotes and ideology to be far more convincing than 
data or research, and, further, some policymakers (particularly lawmakers) outright distrusted the 
research or researchers. One interview participant remarked: 


From a personal standpoint and talking to my peers in other states, the research is helpful, 
but sometimes when the research meets the state legislature or the outside world, it’s not 
always as useful as we'd like it to be. Because I can put the best data in front of legislators and 
if they don’t like it from a policy or personal standpoint, I have to find another way around. 


Maintaining credibility is especially difficult with changing or part-time legislators or when the 
research and data clash with the popular political or ideological positions. The lengthy time periods 
sometimes required to conduct and share research further diminish the ability to be responsive to 
current policy priorities. 


Interview participants also highlighted policymakers’ strong preference for descriptive data gener- 
ated within the state over rigorous evaluations conducted elsewhere—when research is only based on 
external studies, it is often challenging for policymakers to determine whether a program or policy 
would also be beneficial in their own communities. This challenge was especially salient for interview 
participants in Montana, where many institutions and communities face unique challenges related to 
being rural, isolated, and resource constrained, but stakeholders across Indiana and Virginia echoed 
this concern. 


Promising Solutions: See Box 2 for a summary of promising solutions to Barrier 2. Investment in 
relationship-building opportunities between researchers and important decision-makers and stake- 
holders across the state seems to be key for increasing the perceived credibility of research.° For 
example, interview participants discussed the importance of building connections with individual 
lawmakers, particularly across political lines, who can in turn advocate for the use of research and 
data. Furthermore, creating synergy between narratives and research—such as by using research 

to affirm the personal narratives of individual students and, in turn, having students tell their own 
stories related to the research—is a way to use the persuasive power of anecdotes and personal experi- 
ences to support research and to have research support the real-life experiences of students. 


Additionally, policymakers can use state or local data and research in tandem with research from 
outside the state to investigate policy ideas and inform policy implementation. For example, insti- 
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BOX 2 
PROMISING SOLUTIONS TO INCREASE THE CREDIBILITY OF RESEARCH 


O 


Facilitate opportunities for relationship building between researchers and state stakeholders. 


O 


Identify advocates within the legislature to promote the use of research. 


O 


Use outcomes data, focus groups, and small-scale experiments to determine the applica- 
bility and appropriateness of promising innovations from outside the state. 


[-] Develop systems and practices to incentivize local research and experimentation. 


[| Partner with local academics and third-party research organizations to leverage state data 
and external research. 


[_] Researchers: be more responsive to state and organizational culture, ideology, and pri- 
orities. 


tution- or system-level outcomes data can be used to determine whether innovation is appropriate 
for the state, and can create a more compelling case for stakeholders who would otherwise distrust 
the research. VCCS used such a strategy in looking to external research to inform its priorities in 
strengthening academic advising and then using advising data from the community colleges to suc- 
cessfully advocate for additional funding. Further, the state can support local analyses and experi- 
mentation by the development of systems and practices to incentivize research. 


The state agencies involved in Evidence to Action have also partnered with external organizations to 
evaluate policies and programs, which generated rigorous evidence easily applicable to their states. 
According to interview participants, some research partners have also provided support in trans- 
lating research into policy design and implementation. An example of this type of partnership is 
CHE’s and Ivy Tech Community College’s engagement with MDRC’s Scaling Up College Completion 
Efforts for Student Success (SUCCESS) initiative, to implement and sustain comprehensive student 
support programs."° 


Barrier 3: Adopting or scaling evidence-based reforms can be costly or otherwise resource inten- 
sive. Across all states, interview participants stated that the high cost and demand for resources is 
one of the biggest barriers to adopting some prominent reforms backed by rigorous research, such 
as comprehensive student support systems. Additionally, beyond cost, a college president shared 
that his college exhausts significant nonmonetary resources on fundamental challenges outside the 
institution’s control—for instance, a significant proportion of students face extreme poverty and 

the surrounding community has few support services—leaving his institution with little appetite to 
engage in education policy reforms. Finally, many postsecondary institutions do not have the per- 
sonnel necessary to implement reforms. As representatives from VCCS shared, a large proportion of 
postsecondary institutions’ staff and faculty are employed part time and have very limited time and 
energy to engage with innovation or reforms. 
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Promising Solutions: Postsecondary institutions require increased investment and resources to 
incentivize the adoption of rigorously evaluated practice and reform informed by data. (See Box 3.) 
Representatives from SCHEV emphasized the need to provide funding to accompany policy and 
reform whenever possible, rather than placing unfunded or unpaid burdens on institutions and staff 
already operating with constrained resources. An example of such funding is OCHE’s use of left- 
over funding to support state colleges’ pilots of rigorously evaluated interventions; one of these pilots 
is the Montana University System’s Montana 10, a comprehensive support program inspired by the 
proven-effective Accelerated Study in Associate Programs (ASAP), originally designed and imple- 
mented by the City University of New York (CUNY). 


BOX 3 


PROMISING SOLUTIONS TO SUPPORT INSTITUTIONS OPERATING WITH 
CONSTRAINED RESOURCES 


[_] Increase investments in evidence-based success initiatives. 
[_] Provide funding to reduce burdens associated with reform. 


L] Identify creative solutions to incentivize innovation and building evidence. 


Barrier 4: States and postsecondary institutions lack guidance for implementing evidence-based 
reforms and for making the necessary adaptations. Additionally, interview participants highlighted 
the difficulty of building coalitions to set state-level implementation standards and the challenges 
associated with balancing state- and system-wide mandates with local autonomy. State standards can 
support effective statewide implementation: They can support adherence to rigorously tested mod- 
els, facilitate cross-college training and professional development for the institutions and their staff 
members, and simplify the monitoring of institutions’ implementation for continuous improvement. 
Without the assessment of institutions’ progress and success with policy implementation, it is more 
difficult for the state to routinely use prior experiences to inform new initiatives. 


Increasing the tension, institutions also require flexibility to adapt policies to their communities’ 
needs or to facilitate implementation given their available resources and personnel, and the research 
rarely provides enough guidance to inform these decisions without reducing the intervention’s 
impact. For example, interview participants representing institutions serving particular populations 
of students—such as Tribal Colleges and Universities (operated by federally recognized American 
Indian tribes),"* rural institutions, and extremely underresourced institutions—were uncertain 
whether reforms evaluated in more urban or better-resourced contexts were relevant for their com- 
munities or “doable” for their institutions. This lack of clarity, guidance, and support in policy imple- 
mentation can further exacerbate feelings of “initiative fatigue” held by student-facing college faculty 
and staff as they become overwhelmed by multiple large-scale reforms within a short time period. 
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Promising Solutions: Box 4 summarizes promising solutions to make evidence-based reform more 
actionable. Conducting pilot testing, small-scale experimentation, and mentorship practices before 
widespread adoption can provide valuable information about the promise of an innovation to meet 
local needs, strengthen stakeholders’ support for a reform, and generate guidance for later adopters. 
For example, VCCS piloted a developmental education reform by creating cohorts of colleges and set- 
ting up mentorship pairs between colleges belonging to earlier cohorts and colleges in later cohorts. 
Additionally, CHE demonstrates the promise of developing systems for continuous improvement: 
For instance, the commission regularly assesses the effectiveness of its financial aid and workforce 
programs by analyzing student outcomes over time or against a comparison group, produces publicly 
available reports, and uses the findings from these studies to make program improvements. Finally, 
the interview participants emphasized the importance of streamlining and centralizing key func- 
tions (as OCHE did by centralizing some key institutional research functions across the Montana 
University System) to minimize the burden for the colleges. 


BOX 4 


PROMISING SOLUTIONS TO MAKE EVIDENCE-BASED REFORM MORE 
ACTIONABLE 


[_] Conduct pilot testing and experimentation before widespread adoption. 
[-] Centralize the key functions at the state level to minimize the burden on institutions. 


[-] Create standards for implementation and provide institutions with professional development 
and support for executing them. 


[_] Gather and integrate feedback from stakeholders to improve existing reforms. 


[| Analyze and reflect on implementation data and early outcomes for the purposes of continu- 
ous improvement. 


[_] Evaluate existing and new policies and practices to inform developing work. 


[|] Researchers: develop more guidance for implementation and adaptation to maintain the ele- 
ments critical for success while allowing for needed flexibility. 


Overall, two overarching practices emerge from these promising strategies that can be used to sup- 
port the use of research and data in policymaking: 


e Set up long-term partnerships and networks among researchers and policymakers: Maintain- 
ing such alliances allows researchers to stay updated on relevant policy priorities and state poli- 
cymakers to stay aware of emerging opportunities and lessons learned from other states. Studies 
have found that the use of research is best supported when researchers and policymakers create 
personal connections.'* VCCS demonstrates the promise of this practice. For example, the VCCS 
office has strong relationships with researchers, which has helped the community colleges remain 
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deeply engaged in developmental education innovation for over a decade. The other Evidence to 
Action state agency partners also rely heavily on peer networks and personal connections to stay 
informed of national conversations taking place in postsecondary education. 


e Formalize the use of research and data in policymaking systems. Embedding the expectations 
and support for the use of research and data in policymaking systems makes their use routine for 
informing policymaking, rather than relying on individuals’ energy or research literacy.'* It also 
allows states to proactively align their state priorities with available research and data, as is the 
case with CHE’s annual review and evaluation of the state’s workforce-related programs. 


CONSIDERATIONS FOR RESEARCHERS 


For researchers, the interview findings highlighted some other important lessons to consider for 
state-level engagement in higher education: 


e Researchers must take a culturally responsive and strength-based approach to partnerships 
with states. Interview data suggest that there are often cultural misunderstandings between 
researchers and policymakers that dampen trust in the researchers or research. Researchers’ 
investment in relationships that honor and build on the experiences and strengths of important 
stakeholders in postsecondary education—including policymakers, institution leaders, student- 
facing college faculty and staff, and students—will ensure that the research they conduct is 
grounded in the states’ priorities for higher education and accounts for the constraints within 
the state’s policymaking context, as well as increase the credibility granted to the researchers and 
research findings across the state. 


e Research findings—and their intended next steps—should be easily accessible and digestible. 
Given state leaders’ constrained time and comfort with research, interview participants empha- 
sized the importance of researchers highlighting the primary findings and providing attractive 
visuals. It is also important that they identify the major constraints and considerations for inter- 
preting the research findings, identify the remaining open questions that still require further 
study, and clearly articulate the intended next steps for policymakers. As one interview participant 
explained, “With policymakers, the simpler and shorter, the better. Something that’s easy to read 
and understand is important. Relaying personal stories is important, too—[it] helps drive things 
home. The research is important if it can be clear and readable.” 


e There is still a gap between research and actionable guidance. It is essential that rigorous 
studies are accompanied by implementation research, which uses qualitative and quantitative 
studies to understand how an intervention was implemented, the elements that seem critical 
for success, and recommendations to improve future operations. Additionally, intermediaries 
invested in translating research into practice can provide guidance for states and institutions 
interested in adapting interventions to meet their population’s needs and facilitate easier imple- 
mentation that is responsive to the institutional context without compromising effectiveness. 
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MOVING FORWARD 


Similar to many other states, systems, and postsecondary institutions, the Evidence to Action state 
higher education agency partners identified several priority areas for increasing the accessibility 
and value of postsecondary education for their constituents: developmental education, career and 
technical education, academic advising and coaching, and more. Leveraging research, particularly 
the evidence generated by rigorous studies, to support work in these priority areas and incentivizing 
further contributions to the evidence base in these areas are instrumental in advancing students’ 
success in postsecondary education. Additionally, the possible resources available through the 
American Rescue Plan and the College Completion Fund (proposed in the American Families 
Plan) could provide a critical opportunity for states to implement programs with the strongest 
evidence base that would otherwise seem too costly or resource intensive. These opportunities 
reinforce the urgency of providing more guidance and support for states and institutions seeking 
to implement evidence-based practice. 
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